Large language models (LLMs) can store a vast amount of world knowledge, often extractable via question-answering (e.g., "What is Abraham Lincoln's birthday?"). However, do they answer such questions based on exposure to similar questions during training (i.e., cheating), or by genuinely learning to extract knowledge from sources like Wikipedia? In this paper, we investigate this issue using a controlled biography dataset. We find a strong correlation between the model's ability to extract knowledge and various diversity measures of the training data. $\textbf{Essentially}$, for knowledge to be reliably extracted, it must be sufficiently augmented (e.g., through paraphrasing, sentence shuffling) $\textit{during pretraining}$. Without such augmentation, knowledge may be memorized but not extractable, leading to 0% accuracy, regardless of subsequent instruction fine-tuning. To understand why this occurs, we employ (nearly) linear probing to demonstrate a strong connection between the observed correlation and how the model internally encodes knowledge -- whether it is linearly encoded in the hidden embeddings of entity names or distributed across other token embeddings in the training text. This paper provides $\textbf{several key recommendations for LLM pretraining in the industry}$: (1) rewrite the pretraining data -- using small, auxiliary models -- to provide knowledge augmentation, and (2) incorporate more instruction-finetuning data into the pretraining stage before it becomes too late.

Rekor Systems, Inc., a provider of real-time roadway, customer and public safety intelligence to enable AI-driven decisions, announced Tuesday it was selected by the county to implement its Rekor One vehicle recognition system at eight Waukesha County public park entrances. The solution was selected after a competitive bid process earlier this year. Waukesha County Parks utilizes a fee-based system to offset the cost of maintaining thousands of acres of parkland and greenway. According to Rekor, the company's software and hardware will "replace a time-consuming parking enforcement process and support the integration of various entrance fee payment options into one online payment method, better serving both the public and the county." The move comes amid rising popularity of local parks.

WAUKESHA, Wis.--(BUSINESS WIRE)--GE Healthcare today announced the Food and Drug Administration's 510(k) clearance of Critical Care Suite, an industry-first collection of artificial intelligence (AI) algorithms embedded on a mobile X-ray device. Built in collaboration with UC San Francisco (UCSF), using GE Healthcare's Edison platform, the AI algorithms help to reduce the turn-around time it can take for radiologists to review a suspected pneumothorax, a type of collapsed lung. "X-ray – the world's oldest form of medical imaging – just got a whole lot smarter, and soon, the rest of our offerings will too," says Kieran Murphy, President & CEO, GE Healthcare. "GE Healthcare is leading the way in the creation of AI applications for diagnostic imaging and taking what was once a promise and turning it into a reality. By integrating AI into every aspect of care, we will ultimately improve patient outcomes, reduce waste and inefficiencies, and eliminate costly errors. Critical Care Suite is just the beginning."

What makes Netflix so good at predicting movies you'll like? A recommendation algorithm based on your viewing history, and the viewing histories of others like you, of course. While consumer technology is rife with such technology, HarrisData president Lane Nelson sees a future when ERP applications augmented with machine learning algorithms can automate nearly all of the back-office decisions currently made by people. Machines have been taking over the back-office since the days of the typewriter. But the next wave of innovation occurring around big data analytics and machine learning will likely make the transition to a people-less office nearly complete, according to Nelson, who's also holds the title of chief evangelist at HarrisData, the Brookfield, Wisconsin-based provider of enterprise software that runs on IBM i. "Automation is coming at the back office.